Skip to content

Conversation

@RaJiska
Copy link

@RaJiska RaJiska commented Jan 13, 2025

Adds a Nest_Under parameter to the parser filter so that parsed fields can be nested into a field instead of being unpacked in at the root level of the map.

Addresses #8551, and #7775.

Example configuration file:

[SERVICE]
    Parsers_File /fluent-bit/etc/parsers.conf

[INPUT]
    Name  Dummy
    Dummy {"log":"error: my error","element":"{\"a\":\"b\",\"c\":{\"x\":\"y\"}}"}
    Tag   dummy.data

[FILTER]
    Name parser
    Match dummy.*
    Parser json
    Key_Name element
    Nest_Under parsed

[OUTPUT]
    Name stdout
    Match *

This configuration results in the following output:

$ fluent-bit -c dummy.conf
Fluent Bit v2.1.1
* Copyright (C) 2015-2022 The Fluent Bit Authors
...
...
[0] dummy.data: [[1736759501.208000317, {}], {"parsed"=>{"a"=>"b", "c"=>{"x"=>"y"}}}]
[0] dummy.data: [[1736759502.207935361, {}], {"parsed"=>{"a"=>"b", "c"=>{"x"=>"y"}}}]
[0] dummy.data: [[1736759503.207765898, {}], {"parsed"=>{"a"=>"b", "c"=>{"x"=>"y"}}}]

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
    fluentbit-debug.txt

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [N/A] Run local packaging test showing all targets (including any new ones) build.
  • [M/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Backporting

  • [N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@RaJiska
Copy link
Author

RaJiska commented Feb 4, 2025

Hello, in case you missed @edsiper , @leonardo-albertovich , @fujimotos , @koleini . It would be quite useful for us and for others as well I believe.

@RaJiska
Copy link
Author

RaJiska commented Mar 20, 2025

@LukoJy3D
Copy link

+1 from me, as we have cases where nested json gets stringified and can not be easily kept under same key after parsing. Especially needed for Loki when using map keys https://docs.fluentbit.io/manual/pipeline/outputs/loki#structured-metadata-maps

@RaJiska
Copy link
Author

RaJiska commented Aug 7, 2025

Bump @edsiper .

To add more context, this PR will allow field to be nested under a specific key, the purpose of this is to "namespace" parsed results as there may otherwise be conflicts. Take for example a docker app that outputs JSON-formatted logs and that we want to parse, the app timestamping its logs using the time key would conflict with the time already created by through docker logs. Another issue which we face is through Elasticsearch which has a rigid message schema once established, assume app1 having a foo key with a int typed value, app2 also having the foo key but with string value will have its message rejected from Elasticsearch due to conflicting key type which wouldn't happen if app logs were namespace.

@zephinzer
Copy link

Bump

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants